feat(claude): modernize model defaults and bound inline-completion output#389
Open
herikwebb wants to merge 5 commits into
Open
feat(claude): modernize model defaults and bound inline-completion output#389herikwebb wants to merge 5 commits into
herikwebb wants to merge 5 commits into
Conversation
added 5 commits
July 5, 2026 09:08
…tput - Default chat fallback moves to claude-sonnet-5 (current Sonnet tier); the previous claude-sonnet-4-5 default is a generation behind. - Inline completion falls back to claude-haiku-4-5: a suggestion has to beat the user's next keystroke and fires on every pause in typing, so the fastest/cheapest tier is the right default there. - Cap inline-completion max_tokens at 1024 (override with NBI_CLAUDE_INLINE_COMPLETION_MAX_TOKENS); the old 10K ceiling let a rambling response generate for many seconds before the extraction regex threw most of it away. - fetch_claude_models prefers the Models API's max_input_tokens over litellm's static database, which lags new model releases; litellm remains the fallback when the field is absent.
…ride Review follow-up (PR #46): NBI_CLAUDE_INLINE_COMPLETION_MAX_TOKENS was passed to the API unvalidated — a non-integer raised ValueError at import time and blocked the backend from starting, 0/negative values produced failing API requests, and a large value silently restored the unbounded behavior the cap exists to prevent. Parse via _bounded_int_env: malformed values warn and fall back to the 1024 default; out-of-range values are clamped to [1, 4096] with a warning.
Review follow-up (PR #46): the CLAUDE_DEFAULT_* constants name current first-party aliases, but a custom base_url can front an endpoint that serves a different catalog. resolve_default_model prefers the constant only when the endpoint lists it, then falls back to the newest id in the same tier, then to the first listed model. An empty cache (fetch not yet run, or failed) trusts the constant.
Review follow-up (PR #46): resolve_default_model validated defaults against a global cache with no endpoint identity, so after switching to a custom base_url (or while its refresh was in flight) the default could resolve against another endpoint's catalog and send it a model id it doesn't serve. fetch_claude_models now stamps the cache with the normalized (api_key, base_url) it fetched from, and resolve_default_model treats the cache as authoritative only for a matching endpoint — otherwise it trusts the constant until the endpoint's own fetch lands. Credential normalization makes ''/None equivalent, matching the settings-panel empty-field behavior.
Review follow-ups (PR #46, round 4): - The 'newest same tier' fallback sorted model ids lexicographically, which ranks claude-sonnet-4-6 above claude-sonnet-4-10. Sort by the extracted numeric version components instead ([4,6] < [4,10] < [5]), id as tiebreak. - When the cache demonstrably belongs to another endpoint (credentials changed), resolve_default_model now kicks off a background catalog refresh so subsequent resolutions converge on the new endpoint's catalog. It deliberately never blocks on the network: it runs in model constructors on the request path, and a slow or dead custom endpoint must degrade to one possibly-failing request, not a hung chat turn. A never-stamped cache is left to the startup fetch that update_models_from_config already fires.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
claude-sonnet-4-5→claude-sonnet-5(current Sonnet tier).claude-haiku-4-5: a suggestion has to beat the user's next keystroke and fires on every pause in typing, so the fastest/cheapest tier is the right default.max_tokenscapped at 1024 (was 10000), overridable viaNBI_CLAUDE_INLINE_COMPLETION_MAX_TOKENSand clamped to[1, 4096]. The old ceiling let a rambling response generate for many seconds before the extraction regex threw most of it away.resolve_default_modelvalidates the default id against the fetched catalog: if the endpoint doesn't list the preferred alias, it falls back to the newest id in the same tier (numeric version-aware sort, soclaude-sonnet-4-6 < claude-sonnet-5), then to the first listed model. The cache is scoped to the endpoint it was fetched from, and a credentials change triggers a non-blocking background refresh (never blocks a chat turn).fetch_claude_modelsprefers the Models API'smax_input_tokens(validated as a positive int) over litellm's static database, which lags new model releases; litellm remains the fallback.Testing
Full Python suite passes. Manually verified against the live first-party Anthropic API:
max_input_tokens.claude-sonnet-5(matched verbatim).claude-sonnet-99resolves toclaude-sonnet-5(newest in tier), not a lexicographic pick.max_tokensclamp holds end-to-end.Note on a benign startup log
The Models API lists Haiku only under its dated id (
claude-haiku-4-5-20251001), not the rollingclaude-haiku-4-5alias it uses for Sonnet. The inline default therefore isn't matched verbatim and the fallback resolves it to the dated snapshot — the same model, correct outcome. Expected side effect: this INFO line prints on every startup and is not an error:The constant is intentionally left as the rolling alias — resolving it via the fallback is exactly what this change is for.